Language Models

• Trained on large amount of text data
• Statistical models
  • Learn patterns and relationships between words and concepts
• Predict most likely next word based on current sequence

Large Language Models (LLMs)

• Like regular language models, but larger
• Trained on lots of data
• They take up lots of memory (wouldn’t fit on your laptops)

Tokens

• The models are actually built on a concept called “tokens”
  • Parts of words or punctuation
  • Example: “Firefighters save kittens.”
    • “Fire”, “fighters”, “save”, “kittens”, “.”
  • Often there is a unique token for the last word of a text – so it knows when to stop

Embed

• Ultimately, the predictive model is just math
• Math only works on numbers
• Embeddings are like a dictionary that converts each word to a list of numbers
• Similar words will have more similar numbers
  • The numbers for “cat” and “kitten” will be more similar than the numbers for “cat” and “airplane”

Predictions

• The LLM does a ton of math on these numbers to find the most like “next” set of numbers
• It then tries to find the word that has a numerical representation most similar to what it found by the math above, and suggests it

Question-Answering is slightly different than next-word predictions

• Most training texts aren’t in the form of a chatbot
• RLHF - Reinforcement Learning from Human Feedback
  • Extract many possible outputs from the model
  • Get human testers to rank / rate best responses
  • Use that data to improve model
  • Run the loop until they are happy with the results

Why RLHF is useful

Sample Question: What is the capital of France?

Large Language Models are one piece of the puzzle

• The large language model (as described above) only takes in text, and outputs text
  • Related models input and/or output images
  • They have odd-sounding names like gpt-4o-2024-08-08 or claude-3-5-sonnet-20240620 or meta-llama/Meta-Llama-3.1-405B-Instruct-FP8

Most people interact with the applications that were built by OpenAI, et. al.

• OpenAI, Anthropic, and Google have applications that include other features
  • Image readers (usually a different AI model)
  • PDF readers (Theoretically simple)
  • Web Search (Connect to Google or Bing)

These web tools are what people think about

Developers access it through code

Chatbots (Multi-turn)

• The tool sends the entirety of the conversation back to model
• If it gets long enough it will drop the earliest part

Open vs. Closed, API vs Local

• Closed-source: ChatGPT, Claude, Gemini
  • We don’t have access to the processing code - computations are done on their computers
  • On the free versions, they say they may use your inputs for training future models
    • That does not mean that they will spit back your data to future users of ChatGPT - they almost definitely will not
      • Almost definitely ≠ Definitely
    • Can employees at OpenAI read it? 🤷‍♂️

How to keep your data safe? - OpenAI

How to keep your data safe? - Microsoft

• Microsoft Azure OpenAI - API Only
  • Microsoft owns 49% of OpenAI - they claim to run the same API models on their servers
    • Alan’s comment: Not as many features as normal OpenAI, usually delayed
  • NOT THE SAME as Microsoft Co-pilot
    • Microsoft Copilot is their rebranding of Bing’s integration with OpenAI
    • Model is years old, not as good, not as many features

How to keep your data safe? - Local

• There are many open-source models – where you can download the program and control where it runs
  • Most famous is Llama (by Meta/Facebook), but there are many others
• If you try to run them locally, they might produce 15 words a minute, which makes it uselessly slow
  • Unless you have an expensive MacBook Pro
• Generally run on GPUs
  • Can buy local servers, they are expensive to buy and complex to maintain, and get out-of-date quickly

How to keep your data safe? - Cloud

• Amazon Web Services (AWS), Azure (Microsoft), Google Cloud Platform (GCP) are most popular
  • Smaller ones as well, such as RunPod
• High security - they are not looking at your data
  • Every large company has data on AWS / Azure - considered very secure
• Can pay by the hour (as cheap as 40¢/hour)

How to keep your data safe? - Cloud APIs

• Theoretically same as OpenAI, etc… except they don’t train their own models so they’d have no use for your data
• Replit, Modal, Huggingface, many others
• Pay per token (often cheaper that OpenAI)

Prompt Engineering

• Prompt engineering is the process of trying to game the tool by asking a better question
• Input is called “prompt”, output is called “response”
• A generic prompt will usually give a generic response
  • The more information you provide, the better
• Every time you are not happy with a result, add that reason to all future prompts
• Most of these tools can allow for >100k words, so don’t feel you need to conserve - explicate as much as you need

Prompt Engineering - Tips and Tricks

• Role-playing is great
  • Act as a professor at a business school, teaching MBA students about [subject]
• Tell it what to do, not what to avoid
  • It’s not great at negatives

Prompt Engineering - Tips and Tricks

• Use textual borders between sections of your prompt
• Give the subsections titles

Prompt Engineering - Chain of Thought

• Tell it to think step-by-step
  • You can have it check with you each step, or go through it itself

Prompt Engineering - Code the answer

CHATGPT ONLY

• LLMs are notably bad at math or letter-based tasks
• You can ask it to create Python code and run it to get the answer

Few-shot examples

• Giving a few examples of good answers
• Best way to set the optimal format

In-context learning

• The LLM may not have the specific information you need - so tell it!
• Use as much of the box as you need
• Can give it your rubric while grading a project, as well as the specific assignment

RAG

• Build a tool to automatically pull in the correct information
• Combination of search engine & in-context learning
• Breaks the corpus of texts into “chunks”, and identifies which chunks are most likely to answer the question
• Longer contexts mean you can include more options

RAG is the most popular business implementation of LLMs

• Ask questions about your benefit documents
• Ask questions of the US Tax code
• Chat with recent seminars

Fine-tuning

• Actually changes the underlying LLM a bit
• Adding new information
  • Then you don’t have to put in the same few-shot examples or in-context information every time
• Somewhat expensive up-front cost in compute, time, & memory
• You should generally try RAG first, go to the others if there are problems.

Training your own LLM models

DON’T

• Unless you have $50m to spend

Types of AI Models

• Most of the above is text
• There is TTS, STT, Image recognition, Generative Art, Video as well
  • Self driving cars have lots of that
• Integrated multimodal
• Reinforcement learning for game playing
• Protein folding, genomics

Concerns about AI

• Accuracy
• Data Security
• Bias
  • Partisan bias, bigotry, gender, race
• Legality
• Ethics

Build AI models to anonymize data!

• Simpler / cheaper local model can anonymize PII data, so that it can safely be fed into ChatGPT

Prompt Injection

Building Products with AI

• Is risk of error acceptable?
  • What is current human error rate?
• Human-in-the-loop?

Building Products with AI

Issues with Uploading PDFs

If you use AI to generate stuff, people will think its you

Help you on Excel

Help with research

Data to csv

Brainstorming

• Most of its ideas will be bad
• Give it as much context as possible
• Bounce ideas back-and-forth

Create your website

• I don’t know anything about websites, front-end development, or HTML
• Prompting helped me create my website

Critique me

• Give it your lecture notes

You are a First-Year MBA Student. What about my lecture did you not understand?

• Give it your paper

You are peer-reviewer #2, who is a harsh critic and will reject my paper. What reasons will you give?

Tools / Software

• CircleBack
  • Records meetings, takes notes, identifies to-do items
  • Integrates with Zoom, Team, Slack

Tools / Software

• Gamma
  • Pretty presentations, documents, websites

Tools / Software

• Readwise
  • Personal Librarian
  • Can tell you where within documents to read
  • Can find documents from months ago

Chat with your documents

• Every company has a document management problem
  • e.g. Sell-side analyst reports
  • Build compare & contrasts
• Every industry needs that - including government

Finance

• Chat with your investments (my company 😃)
  • Which sector has my biggest concentration of risk?
  • How does [financial goal X] fit in with my current portfolio?

Finance - Deal Analysis

• Private Equity firm getting access to huge folder of data
  • LLMs can understand underlying company
  • Extract data, put it into model
  • Identify information e.g. competitors, risks, etc…

Automatic report writing

• Assume you have software (or can create that assesses quantitative/qualitative risks)
  • Automatically describe risks (you can guide it how technical can be)
  • Automate the in-context learning to ensure it doesn’t lie
• Every industry needs that

Lobbying - Market Research

• Government regulations needs to be posted online - with open comments
• Understand all the comments - pro & con
• LLM can aggregate arguments -> Better understanding other guides how you can frame your arguments

Pharma

• It can take 4 months and > $200k to develop the templates for clinical trial intake information
• LLMs can prepare the information for a human pre-review and save ~ 1/3 of the time

Medical Billing

• Match patient chart to medical codes
• Currently done by low-wage offshore employees
• NEED TO BE CAREFUL ABOUT HIPAA

Lightning Rod Installation

• Months behind on goals to create project estimates
• 500 Page PDFs with architectural drawings
• Often looking for specific location on one specific chart
• Alternatively, need to identify any protrusions that might cause issues
• Combine text LLM & vision LLM

3D Printing

• 3D printers are cool, but creating correctly specs is difficult
• Tool - describe what you want, output a correctly specified .CAD file and an image of possible output

Interviewer practice

• For certain types of difficult interview (e.g. asylum seekers), simulate interviewee, and inteviewers can practice
• Automatically grade
  • In terms of presentation (did they balance politeness and firmness)
  • Content
    • Did they identify lies?
    • Did they extract all necessary/relevant information

Education

• Licensing bodies (e.g. CPA, CFA)
  • Audio versions of materials
  • Training materials more specified to certain needs
  • Continuing education

Health Insurance - Information matching

• Address matching (e.g. “123 W 4th St” vs “123 West Fourth Street”)
• Policy matching (internal and external document might have different names for policies)

Interior Design

• Feed a picture of your room, and describe how you’d want to change it
• Output image of room – give it critiques
• Identify stores / contractors who can make it happen

College advising

• You can say what you want in a college (e.g. good 19th century american history, near nature, small class size, etc…)
• Output list of best colleges for you

Auto navigation

• “What is a good mexican restaurant on my route?”

Restaurant assistants

• You can take a picture of your supplier receipts, upload your menu, and show the exact costs that go into each dish

Government - Immigration

• Automatically process immigration forms

Further Reading

Contact me for more questions

alanfeder@gmail.com